Real-Time Data Mining of Massive Data Streams from Synoptic Sky Surveys

摘要

The nature of scientific and technological data collection is evolvingrapidly: data volumes and rates grow exponentially, with increasing complexityand information content, and there has been a transition from static data setsto data streams that must be analyzed in real time. Interesting or anomalousphenomena must be quickly characterized and followed up with additionalmeasurements via optimal deployment of limited assets. Modern astronomypresents a variety of such phenomena in the form of transient events in digitalsynoptic sky surveys, including cosmic explosions (supernovae, gamma raybursts), relativistic phenomena (black hole formation, jets), potentiallyhazardous asteroids, etc. We have been developing a set of machine learningtools to detect, classify and plan a response to transient events for astronomyapplications, using the Catalina Real-time Transient Survey (CRTS) as ascientific and methodological testbed. The ability to respond rapidly to thepotentially most interesting events is a key bottleneck that limits thescientific returns from the current and anticipated synoptic sky surveys.Similar challenge arise in other contexts, from environmental monitoring usingsensor networks to autonomous spacecraft systems. Given the exponential growthof data rates, and the time-critical response, we need a fully automated androbust approach. We describe the results obtained to date, and the possiblefuture developments.